{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tool Call Accuracy Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates how to use tool call accuracy evaluator on agent data. The supported input formats include:\n",
    "- simple data such as strings and `dict` describing tool calls;\n",
    "- user-agent conversations in the form of list of agent messages. \n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Tool Call Accuracy evaluator assesses how accurately an AI uses tools by examining:\n",
    "- Relevance to the conversation\n",
    "- Parameter correctness according to tool definitions\n",
    "- Parameter value extraction from the conversation\n",
    "- Potential usefulness of the tool call\n",
    "\n",
    "The evaluator uses a binary scoring (0 or 1) for each tool call:\n",
    "\n",
    "    - Score 0: The tool call is irrelevant or contains information not in the conversation/definition\n",
    "    - Score 1: The tool call is relevant with properly extracted parameters from the conversation\n",
    "\n",
    "If there are multiple call, the final score will be an **average** of individual tool calls, which can be interpreted as the **passing rate** of tool calls.\n",
    "\n",
    "This evaluation focuses on measuring whether tool calls meaningfully contribute to addressing query while properly following tool definitions and using information present in the conversation history."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tool Call Accuracy requires following input:\n",
    "- Query - This can be a single query or a list of messages(conversation history with agent). Latter helps to determine if Agent used the information in history to make right tool calls.\n",
    "- Tool Calls - Tool Call(s) made by Agent to answer the query. Optional - if response has tool calls, if not provided evaluator will look for tool calls in response.\n",
    "- Response - (Optional) Response from Agent (or any GenAI App). This can be a single text response or a list or messages generated as part of Agent Response. If tool calls are not provide Tool Call Accuracy Evaluator will look at response for tool calls.\n",
    "- Tool Definitions - Tool(s) definition used by Agent to answer the query. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Tool Call Accuracy Evaluator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation import ToolCallAccuracyEvaluator, AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "\n",
    "tool_call_accuracy = ToolCallAccuracyEvaluator(model_config=model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating Single Tool Call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle ?\"\n",
    "tool_call = {\n",
    "    \"type\": \"tool_call\",\n",
    "    \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"arguments\": {\"location\": \"Seattle\"},\n",
    "}\n",
    "\n",
    "tool_definition = {\n",
    "    \"id\": \"fetch_weather\",\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tool_call_accuracy(query=query, tool_calls=tool_call, tool_definitions=tool_definition)\n",
    "pprint(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Multiple Tool Calls used by Agent to respond"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle ?\"\n",
    "tool_calls = [\n",
    "    {\n",
    "        \"type\": \"tool_call\",\n",
    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"arguments\": {\"location\": \"Seattle\"},\n",
    "    },\n",
    "    {\n",
    "        \"type\": \"tool_call\",\n",
    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"arguments\": {\"location\": \"London\"},\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definition = {\n",
    "    \"id\": \"fetch_weather\",\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tool_call_accuracy(query=query, tool_calls=tool_calls, tool_definitions=tool_definition)\n",
    "pprint(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tool Calls passed as part of `Response` (common for agent case)\n",
    "- Tool Call Accuracy Evaluator extracts tool calls from response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Can you send me an email with weather information for Seattle?\"\n",
    "response = [\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:37Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"weather\": \"Rainy, 14\\u00b0C\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:38Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
    "                \"name\": \"send_email\",\n",
    "                \"arguments\": {\n",
    "                    \"recipient\": \"your_email@example.com\",\n",
    "                    \"subject\": \"Weather Information for Seattle\",\n",
    "                    \"body\": \"The current weather in Seattle is rainy with a temperature of 14\\u00b0C.\",\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:41Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"tool_result\", \"tool_result\": {\"message\": \"Email successfully sent to your_email@example.com.\"}}\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:42Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle. The current weather is rainy with a temperature of 14\\u00b0C.\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tool_call_accuracy(query=query, response=response, tool_definitions=tool_definitions)\n",
    "pprint(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Response as String (str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Book a flight to New York for tomorrow\"\n",
    "\n",
    "# Response as a simple string instead of a list of messages\n",
    "response = \"I've found several flight options to New York for tomorrow. I'll use the booking tool to reserve your seat.\"\n",
    "\n",
    "tool_calls = [\n",
    "    {\n",
    "        \"type\": \"tool_call\",\n",
    "        \"tool_call_id\": \"call_book_flight_456\",\n",
    "        \"name\": \"book_flight\",\n",
    "        \"arguments\": {\n",
    "            \"destination\": \"New York\",\n",
    "            \"date\": \"tomorrow\"\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"book_flight\",\n",
    "        \"description\": \"Books a flight to the specified destination on the given date.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"destination\": {\"type\": \"string\", \"description\": \"The destination city.\"},\n",
    "                \"date\": {\"type\": \"string\", \"description\": \"The date of the flight.\"},\n",
    "            },\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "result = tool_call_accuracy(query=query, response=response, tool_calls=tool_calls, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Response as List[dict] with Tool Definition as Single Dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What's the weather in San Francisco?\"\n",
    "\n",
    "response = [\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T18:15:22Z\",\n",
    "        \"run_id\": \"run_abc123\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_sf_weather_789\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"San Francisco\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T18:15:24Z\",\n",
    "        \"run_id\": \"run_abc123\",\n",
    "        \"tool_call_id\": \"call_sf_weather_789\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"weather\": \"Foggy, 18°C\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T18:15:25Z\",\n",
    "        \"run_id\": \"run_abc123\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"The weather in San Francisco is currently foggy with a temperature of 18°C.\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definition_dict = {\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "    },\n",
    "}\n",
    "\n",
    "result = tool_call_accuracy(query=query, response=response, tool_definitions=tool_definition_dict)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Query as Conversation History (List of Messages)\n",
    "The evaluator also supports query as a list of messages representing conversation history. This helps determine if the Agent used the information in the conversation history to make the right tool calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Query as conversation history instead of a single string\n",
    "query_as_conversation = [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\", \n",
    "        \"content\": \"Hi, can you check the weather in Seattle for me?\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": \"Actually, could you also send me an email with that weather information to john@example.com?\"\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_calls = [\n",
    "    {\n",
    "        \"type\": \"tool_call\",\n",
    "        \"tool_call_id\": \"call_weather_123\",\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"arguments\": {\"location\": \"Seattle\"},\n",
    "    },\n",
    "    {\n",
    "        \"type\": \"tool_call\", \n",
    "        \"tool_call_id\": \"call_email_456\",\n",
    "        \"name\": \"send_email\",\n",
    "        \"arguments\": {\n",
    "            \"recipient\": \"john@example.com\",\n",
    "            \"subject\": \"Weather Information for Seattle\",\n",
    "            \"body\": \"Here is the weather information you requested.\"\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tool_call_accuracy(query=query_as_conversation, tool_calls=tool_calls, tool_definitions=tool_definitions)\n",
    "pprint(response)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
